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Shipyard  Trade  Skill  Testing  Program 

John  Walker  Hartigan,  Visitor,  Naval  Sea  Systems  Command 


The  eight  naval  shipyard,  in  conjunction  with  the 
Naval  Sea  Systems  Command  have  developed  a  system 
of  written  and  performance  tests  applicable  to 
journeymen  level  production  workers.  These  tests  were 
developed  in  compliance  with  the  government '' s  Uniform 
Guidelines  on  Selection  Procedures  and  submitted  to  tbe 
U.S.  Office  of  Personnel  Management  for  approval. 
Four  specific  applications  are  envisioned  for  tbe  tests:  (1) 
Promotion  of  worker-grade  personnel  to  journeyman 
positions;  (2)  Hiring  experienced  personnel  from  outside 
the  federal  government;  (3)  Identification  of 
reguirements  for  either  additional  or  remedial  training 
of  apprentices;  and  (4)  Identification  of  deficiencies  in 
personnel  training  programs.  Growth  plans  for  tbe 
programs  include  progress  test  for  apprentice  programs, 
measurement  associated  with  qualifications  on  new 
procedures,  and  certification  in  lieu  of  training  on  CCT- 
tain  repetitive  programs. 

Tests  have  been  developed  for  seventeen 
trades  and  validated  by  trial  administrations  at  each 
of  the  naval  ship  yards.  As  part  of  the  development 
process  a  task  analysis  was  performed  for  each  of  the 
trades.  Computer  programs  used  for  test  generation 
allow  the  test  designer  to  specify  which  tasks  are  to 
be  measured  in  any  given  test  and  allow  the  tests  to 
accommodate  differences  in  job  content  and 
procedures  at  the  several  locations..  Scoring  and 
analysis  features  are  also  included  in  the 
computerized  testing  programs. 

The  task  analysis  and  the  computer  programs 
are  in  the  public  domain.  As  such  they  can  be  made 
available  to  commercial  shipyards  with  potential  test¬ 
ing  applications. 

NOMENCLATURE 

The  following  acronyms  and  abbreviations  are 
used  in  this  paper. 


SSTS 

T  S  D 

USTIS 

S  D 

OPM 

FPM  335-1 

0  C  P  M 

INTRODUCTION 


Shipyard  Skills-Tracking 
System 

Trade-Skill  Designator 

Uniform  Shipyard 
Training  Information 
System 

Supervisor's  Desk 

u.  s.  office  of 
Personnel  Management 

Federal  Personnel 
Manual  Supplement 
335-1 

Department  of  the  Navy 
Office  of  Civilian 
Personnel  Management 


The  Naval  Sea  Systems  Command 
Headquarters  (NAVSEA)  is  working  with  the  eight 
naval  shipyards  to  develop  batteries  of  written  and 
performance  tests  that  can  be  used  to  measure  pro¬ 
duction  worker's  capabilities  of  accomplishing  as¬ 
signed  shipwork.  The  tests  are  intended  to  help 
shipyards  to  assign  the  right  personnel  to  complex 
jobs;  to  determine  the  best  sections  within  each  trade 
to  assign  newly-hired  personnel;  to  determine  what 
specialized  training  trade  personnel  require;  to  assess 
the  efficacy  of  trade  training  programs;  and,  upon 
approval  by  the  U.S.  Office  of  Personnel 
Management  (OPM) ,  to  make  informed  decisions  in 
selection  and  promotion  of  trade  personnel. 


NAVSEA 


Naval  Sea  Systems  SCENARIO 

Command  Headquarters 


NAVSHI IP '  YD  SJOSE 


San  Jose  Naval  Shipyard 
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The  scene  is  just  a  short  time  from  now  at  the 
(fictitious)  San  Jose  Naval  Shipyard  (NAVSHIPYD 
SJOSE) .  A  project  team  has  just  assembled  at  its 


production  worksite  to  prepare  to  accomplish  its  first 
assigned  job  on  an  SSN-688-class  submarine  refuel 
ing  overhaul.  The  members  have  been  selected  be 
cause  the  automated  Shipyard  Skills-Tracking 
System  (SSTS)  [already  in  operation  in  seven  of  the 
eight  real  naval  shipyards]  has  shown  that  they 
possess  the  appropriate  Trade-Skill  Designators 
(TSDs)  [under  active  development  in  the  real  world] 
and  that  they  possess  the  requisite  formal 
qualifications  to  perform  a  number  of  the  tasks  their 
assigned  job  will  entail. 

Despite  their  qualifications,  however,  this  job 
will  involve  accomplishment  of  a  number  of  tasks 
that  none  of  them  has  ever  performed;  some  of  these 
tasks  are  so  sensitive  to  error  (and  improper 
performance  would  be  so  expensive  to  correct)  that 
the  team-and  the  shipyard-need  to  be  absolutely 
certain  that  those  who  perform  them  are  fully 
competent  to  do  so.  In  addition,  none  of  the  team 
members  has  ever  worked  a  job  before  following  the 
precepts  of  zone  technology,  nor  have  any  of  them 
worked  outside  the  boundaries  of  the  specific  trades 
in  which  they  were  trained.  Working  in  this  manner 
will  entail  a  significant  cultural  change  for  them. 

a.  Having  rounded  up  and  reviewed  all  of  the 
required  technical  .documentation  for  the  job,  the 
team  begins  to  parcel  out  work  assignments  to  its 
individual  team  members.  Willie,  the  team  leader 
says  to  Joe,  an  electronics  mechanic;  “Joe,  you 
clearly  have  the  best  background  to  perform  Task 
03.4.  And  the  job  will  be  performed  most  efficiently 
if  you  also  perform  associated  tasks  03.3  and  03.5. 
But  you’ve  never  performed  those  specific  tasks 
before.  So  here’s  what  we’re  going  to  do.’’ 

b.  Willie  turns  to  a  video  work  station  that 
has  been  brought  to  the  worksite.  When  she  touches 
the  screen,  a  menu  pops  up.  From  the  menu,  she 
selects  “enter  task  assignments.”  Following  the 
directions  on  the  screen,  she  enters  Joe’s  badge 
number  and  task  numbers  03.3,  4,  and  5.  From  a 
menu,  she  then  selects  “task  training  and  testing” 
and  selects  task  number  03.3  as  the  starting  point. 
Turning  back  to  Joe,  she  says  “Go  for  it!” 

c.  Joe  sits  down  at  the  work  station,  which  con¬ 
tains  a  microcomputer  containing  a  702-megabyte 
hard  drive  capable  of  storing,  digitally,  as  many  as 
72  minutes  of  full-motion  video  (or  combinations  of 
motion  video,  still  video,  computer  graphics, and 
text).  Upon  pressing  the  “enter”  key,  he  is  adminis 
tered  a  brief  pre-test  of  what  he  knows  about  how  to 
perform  task  03.3.  The  pre-test  reveals  that,  while 
Joe  is  well-versed  in  most  aspects  of  the  task,  he  is 
unfamiliar  with  a  couple  of  the  specifications  he 
must  meet  in  accomplishing  the  task 

d.  To  assure  that  Joe  will  have  the  requisite 
knowledge  to  perform  task  03.3,  thevideo/computer 
system  administers  a  tutorial  to  him  on  the 


knowledge  requirements  of  task  03.3,  with  test 
questions  built  into  the  lesson.  Joe  views  the  lesson, 
answering  the  questions  as  they  come  up  by  touching 
the  choices  listed  on  the  screen.  When  he  picks  a 
wrong  answer,  the  system  sends  him  back  to  review 
the  part  of  the  lesson  containing  the  material  he  failed 
to  master.  If  Joe  misses  the  right  answer  to  the  same 
test  question  a  second  time,  he  is  sent  back  to  review 
the  problem  material  once  again-but  this  time,  the 
content  is  presented  to  him  in  a  different  way,  on  the 
assumption  that  there  was  something  about  its 
presentation  the  first  two  times  that  didn’t  fit  Joe’s 
learning  style.  If  the  test  question  measures 
knowledge  that  is  essential  to  correct  performance  of 
task  03.3,Joe  is  not  permitted  to  progress  through  the 
lesson  until  he  has  demonstrated  that  he  has  acquired 
that  knowledge 

e.  Once  having  completed  the  lesson  for  task 
03.3,  the  interactive-multimedia  system— for  that,  in 
fact,  is  the  current  name  for  the  kind  of  system  Joe  is 
training  on— moves  Joe  into  a  pre-test  for  task  03.4. 
Because  the  shipboard  system  on  which  Joe  will  be 
working  is  so  sensitive  to  error,  it  is  essential  that  task 
03.4  be  performed  correctly  in  every  detail;  the 
passing  grade  for  the  lesson  on  task  03.4  is  therefore 
set  at  100  percent  Joe,  however,  is  experienced  at 
performing  the  task  and  answers  all  the  questions  in 
the  lesson  correctly  the  first  time.  The  system 
therefore  notifies  the  Shipyard  Skills-Tracking  System 
(SSTS)  to  update  Joe’s  qualificaion  records  to  show 
that  he  has  requalified  for  task  03.4  as  of  this  date, 
and  it  moves  Joe  into  the  lesson  for  task  03.5. 

f.  Task  03.5  is  an  installation  process  that  re¬ 
quires  performance  of  a  number  of  work  steps  in  a 
precise  sequence.  Joe  is  therefore  shown  a  video  of  a 
mechanic  performing  the  task;in  the  video,  the  key 
work  steps  are  shown  in  slow  motion  and  repeated 
several  times,  while  the  audio  describes  the  steps  be¬ 
ing  performed  and  why  it  is  important  that  they  be 
performed  as  shown.  Joe  is  then  tested  by  being 
shown  a  video  of  a  mechanic  performing  the  task  at 
normal  speed;  Joe  is  asked  to  touch  the  screen  if  he 
sees  a  mistake  being  made.  He  is  next  required  to 
walk  through  a  simulation  of  the  task,  performing  all 
of  its  worksteps,  in  sequence,  by  touching  the  objects 
shown  on  the  screen  at  the  appropriate  times. 

g.  Joe  successfully  completes  the  simulation  of 
task  03.5  and  reports  to  his  team  leader.  The  team 
leader  goes  to  the  workstation,  identifies  herself  to  the 
system,  and  ask  for  a  report  on  Joe’s  performance. 
The  system  displays  Joe’s  grades  on  the  lessons  for 
task  03.3,  03.4,  and  03.5,  including,  if  the  team 
leader  ask  for  it,  how  many  attempts  Joe  required  on 
each  question  or  each  lesson  before  passing  it. 
Satisfied  that  Joe  is  fully  prepared  to  accomplish  the 
three  task,  the  team  leader  directs  Joe  to  begin  work. 

h.  When  Joe  gets  to  task  03.5,  he  wants  to  make 
sure  he  performs  its  complicated  work  steps  in  the 
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sequence  demanded.  He  turns  to  the  work  station 
and  calls  up  the  task.  He  then  watches  the  mechanic 
on  the  screen  perform  each  step;  puts  the  system  on 
“pause”;  mimics  on  the  real  ship  component  what  he 
has  just  seen;  restarts  the  system  and  repeats  the 
process  with  each  succeeding  work  step,  thereby 
ensuring  that  he  has  left  nothing  to  memory  or 
chance. 

i.  Upon  completion  of  the  three  task  (and  of 
any  post-performance  inspections  required),  Joe  in¬ 
serts  his  badge  into  a  slot  in  the  work  station.  This 
causes  real-time  work-status  updates  in  Supervisor’s 
Desk  [currently  under  development  in  the  real 
world]  and  in  the  shipyard’s  central  management -in¬ 
formation  system  modules;  it  also  further  updates 
Joe’s  qualification  and  experience  records  in  SSTS 
and  his  training  records  in  the  Uniform  Shipyard 
Training  Information  System(USTIS)  [another 
system  that  is  currently  under  development]. 

DISCUSSION 

The  foregoing  scenario  describes  how  “just-in 
time”delivery  of  training  could  work  in  a  shipyard. 
Some  of  the  dividends  shipyards  would  reap  are 
obvious,  such  as  virtual  elimination  of  the  problem 
of  mental  retention  of  knowledge  and  skills  imparted 
through  training.  Other  benefits  might  not  be  as 
readily  apparent,  such  as  the  fact  that  the  blurring  of 
the  distinction  between  training,  on  the  one  hand, 
and  job  preparation  and  performance,  on  the  other, 
will  likely  make  it  possible  for  much  of  this  training 
to  be  charged  directly  to  ship  customers,  rather  than 
to  overhead. 

Testing  plays  an  intimate  and  prominent  role 
in  this  kind  of  training.  Testing  is  integral  to  the 
process  of  preparing  personnel  to  accomplish  ship 
repair  work  and  to  determining  the  relative  compe 
tencies  of  those  available  to  dot  he  work  The  repair 
of  modem  naval  ships  and  submarines  has  become  so 
sophisticated,  and  the  consequences  of  error  so 
significant,  that  to  leaving  to  chance  the  capabilities 
of  those  performing  ship  repairs  and  alterations. is 
not  acceptable  NAVSEA  through  its  Trade-Skill 
Testing  Program  is  therefore  making  available  to  the 
naval  shipyards  validated  written  and  performance 


tests  to  assess  the  job  proficiency  of  its  wage-grade 
workers  in  these  skilled  trades.  Occupational  tests- 
i.e.,  tests  that  assess  whether  those  tested  are  capable 
of  performing  as  journeyman  mechanics  in  their  re¬ 
spective  trades— have  been  issued  for  use  in  the  pro¬ 
motion,  selection,  placement,  and  training  of  shipyard 
employees  in  17  major  skilled  trades.  The  tests  may 
not  be  used  in  promotions  or  in  selection  of 
applicants  from  outside  the  testing  shipyard  without 
the  tests  having  first  been  approved  for  these  uses  by 
OPM,  a  process  that  is  underway.]  The  Ii’  trades 
covered  are  shown  in  figure  1. 

A  Trade-Skill  Testing  Program  Users’  Manual 
has  been  issued  to  the  naval  shipyards.  It  contains 
guidelines  that  are  designed  to  be  consistent  with 
current  federal  and  Navy  regulations  governing  the 
selection  and  promotion  of  employees,  such  as  FPM 
335-1.  It  should  be  emphasized  that  these  are 
guidelines,  not  mandatory  requirements.  Naval 
shipyards  have  the  option  of  deviating  from  the 
guidelines  as  long  as  their  deviations  are  consistent 
with  current  Navy  and  federal  personnel  regulations. 
A  companion  manual,  the  Examiner’s  Manual, 
provides  detailed  instructions  for  administering  and 
scoring  of  the  tests. 

The  trade-skill  tests  have  been  validated  under 
the  requirements  of  FPM  335-1  and  have  been. ap¬ 
proved  by  NAV SEA  for  use  in  the  naval  shipyards 

under  delegated  authority  from  Navy’s  Office  of 
Civilian  Personnel  Management  (OCPM). 

The  occupational  examinations  may  be  used  in 
the  following  ways: 

1.  To  promote  current  shipyard  workers 
(apprentices  and  “limited”  workers)  to 
journeyman-level  jobs. 

2.  To  select  and  place  external  applicants  with 

trade  experience  in  journeyman-level  jobs. 

3.  To  determine  the  training  needs  of  the  pre 

sent  workforce. 

4.  To  evaluate  the  effectiveness  of  apprentice 

and  other  training  programs. 

It  is  important  to  note  that  NAVSEA  has  laid 
down  the  policy  that  the  tests  are  not  to  be  used  for 
adverse  personnel  actions,  such  as  firing  and  demo¬ 
tions. 

Boilermaker 
Electronics  Mechanic 
Fabric  Worker 
Inside  Machinist 
Outside  Machinist 
Pipefitter 

Sheetmetal  Mechanic 
Shipwright 


Air-conditioning  Equipment  Mechanic 
Electrician 
Electroplater 
Insulator 

Ordnance  Equipment  Mechanic 
Painter 
Rigger 
Shipfitter 
Welder 

Figure  1.  The  17  trades  for  which  occupational  tests  have  been  developed. 
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NATURE  OF  THE  TESTS 

The  set  of  tests  for  each  trade  consists  of  a  mul¬ 
tiple-choice  written  examination  and  a  “hands-on” 
performance  test.  This  was  done,  in  part,  because 
the  U.S.  Office  of  Personnel  Management  will  not 
permit  use  of  written  examinations  alone  in  selection 
and  promotion  actions.  (This  policy  stems  from 
evidence  that  written  tests,  when  used  alone,  may 
have  an  adverse  impact  upon  socio-economically 
disadvantaged  applicants 

The  written  tests  in  each  trade  consist  of  100  to 
150  multiple-choice  questions  that  assess  trade 
knowledge  and  ability  to  apply  such  knowledge.  The 
tests  take  approximately  two  hours  each  to  ad 
minister. 

The  performance,  or  practical,  tests  measure  job 
skills,  using  actual  tools  and  equipment  of  the  trade. 
The  performance  tests  take  approximately  one  to 
four  hours  to  administer. 

DEVELOPMENT  OF  THE  TESTS 

Professional  test  designers  were  hired  to  shep 
herd  development  of  the  tests.  This  was  done  to 
ensure  that  the  tests  maintained  both  face  validity 
and  content  validity;  that  every  test  item  would  be 
validated  against  essential  job  requirements;  and  that 
the  tests  met  the  standards  of  the  federal  gov¬ 
ernment’s  Uniform  Guidelines  on  Employee 
Selection  Procedures.  This  kind  of  rigor  was  neces 
sary  to  the  securing  of  OPM  approval  of  the  tests 
and  to  precluding  grievances  and  litigation.  If 
grievances  or  litigation  against  the  tests  should  occur, 
the  tests  can  be  defended  on  the  basis  that  they 
measure  knowledge  and  skills  that  are  essential  to 
competent  job  performance. 

The  tests  were  developed  by  panels  of  repre¬ 
sentatives  of  the  17  trades,  guided  by  the  professional 
test  designers.  As  part  of  the  development  process,  a 
task  analysis  was  performed  for  each  of  the  trades. 
This  analysis  divided  the  trade  into  several  major 
duties;  identified  tasks  within  the  duties;  and 
identified  the  knowledge  and  skills  required  for  a 
worker  to  perform  each  task  competently  at  the 
journeyman  level.  (The  task  analyses  were  designed 
to  accommodate  differences  in  procedures  and  job 
content  within  the  same  trade  among  the  eight  naval 
shipyards.) 

Each  of  the  written  test  items  is  tied  to  the  task 
analysis  and  has  also  been  referenced,  wherever  pos 
sible,  to  applicable  technical  specifications. 

Once  written,  the  test  items  were  loaded  into  a 
computerized  test-item  data  bank.  Computer  pro¬ 
grams  have  been  written  that  enable  semi-automated 
test  generation:  the  test  designer  has  merely  to 
specify  which  task  are  to  be  measured  in  a  given  test, 


and  the  computer  assembles  appropriate  test  items 
into  a  new 

test.  Other  computer  programs  include  automated 
test  analysis  and  scoring  of  test  answer  sheets. 

All  of  the  tests  were  administered  on  a  trial  basis 
to  journeymen  at  the  naval  shipyards  and  were  revised 
to  ensure  that  they  are  reliable,  fair,  and  valid 
measures  of  trade-skill  and  knowledge. 

CONCLUSION 

Following  implementation  of  the  initial  17 
occupational  trade-skill  tests,  NAVSEA  plans  to  begin 
development  of  two  additional  kinds  of  tests: 

Progress  Tests 

Progress  tests  will  be  developed  to  be  adminis¬ 
tered  to  apprentices  every  six  months,  as  they  become 
eligible  for  their  semiannual  promotions.  These  tests 
will  give  shipyards  a  clearer  picture  of  the  efficacy  of 
their  apprentice  training  programs  and  should  also 
serve  to  spur  production  shops  toward  greater 
conscientiousness  in  managing  the  rotation  of 
apprentices  among  the  skills  of  their  trades. 

Specialized  Tests 

Specialized  tests  in  advanced  technical  skills. will 
be  developed.  Examples  of  such  tests  include  tests  of 
mechanics’  ability  to  perform  jobs  never  before 
accomplished  at  a  particular  shipyard;  tests  of  ability 
to  perform  the  work  involved  in  a  complex  ship 
alteration  (ShipAlt);  and  tests  of  competence  on  jobs 
that  have  been  generating  rework.:and,  of  course,  tests 
to  be  integrated  into  the  fabric  of  interactive  work 
qualification  and  assignment  process  described  in  the 
scenario  that  opened  this  paper. 

ACKNOWLEDGEMENT 

The  author  was  assisted  in  the  preparation  of 
this  paper  by  Lieutenant  Commander  C.  Lee  Walker, 
USN  (Retired). 

References: 

1  Trade-Skill  Test  Users'  Manual  (draft), 
Washington,  D.C.,  Naval  Sea  Systems 
Command,  June  1990. 

2.  Uniform  Guidelines  for  Employee  Selection 
Procedures,  FPM  Supplement  271-2, 

Washington,  D.C.,  United  States  Office  of 
Personnel  Management,  August  31,  1979. 


IXB2-4 


Additional  copies  of  this  report  can  be  obtained  from  the 
National  Shipbuilding  Research  and  Documentation  Center: 

http://www.nsnet.com/docctr/ 

Documentation  Center 
The  University  of  Michigan 
Transportation  Research  Institute 
Marine  Systems  Division 
2901  Baxter  Road 
Ann  Arbor,  Ml  48109-2150 

Phone:  734-763-2465 

Fax:  734-936-1081 

E-mail:  Doc.Center@umich.edu 


